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Executive  Summary 
Objectives 

The  Government  of  Alberta  has  introduced  the  concept  of  three  year  business  plans  for  its 
departments,  agencies,  boards  and  commissions  as  part  of  its  strategy  to  improve  public 
services  within  budget  targets.  Alberta  Education's  plan,  Meeting  the  Challenge  II  -Three-Year 
Business  Plan  1995/96  to  1997/98,  provides  direction  for  the  future  of  education.  The  plan 
calls  for  better  reporting  of  educational  information  to  enhance  accountability  to  the  public  for 
results  achieved.  To  this  end,  Alberta  Education  commissioned  a research  study  to  collect 
information  on  the  satisfaction  levels  of  various  stakeholders  with  the  education  system.  This 
research  involved  3611  telephone  interviews  with  seven  different  stakeholder  groups  between 
March  8 and  April  7,  1995. 

In  addition  to  this  provincial  research,  it  is  anticipated  that  similar  types  of  satisfaction  surveys 
will  be  conducted  by  individual  schools  and  school  jurisdictions  across  the  province  in  the 
coming  years.  Consequently,  it  was  decided  to  conduct  a pilot  project  that  would  have  the 
following  objectives: 

To  compare: 

■ the  results  of  the  provincial  telephone  survey  with  survey  results  at  the  school 
level. 

■ the  results  from  telephone  interviews  with  the  results  from  self  administered 
questionnaires  (the  methodology  most  likely  to  be  used  by  schools  and  school 
jurisdictions). 

To  assess  local  training  requirements  that  schools/school  jurisdictions  have  for  conducting  self 
administered  and/or  telephone  interviews. 

Methodology 

The  specific  research  methodology  for  the  pilot  project  was  as  follows: 

1.  For  each  school  jurisdiction,  one  school  was  selected  to  participate  in  the  study. 

2.  School  staff  and  system  administration  prepared  a questionnaire  for  each  school  that 
included  some  key  questions  that  were  asked  on  the  provincial  survey.  In  addition,  each 
school/  school  jurisdiction  included  a list  of  questions  that  probed  issues  on  which  more 
information  was  wanted,  but  were  not  part  of  the  provincial  survey.  Three  separate 
questionnaires  were  prepared  for  each  school/  school  jurisdiction:  one  for  parents,  one 
for  high  school  students,  and  one  for  teachers. 
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3.  Each  school  jurisdiction  prepared  a written/  self-completion  questionnaire,  and  one  that 
could  be  administered  over  the  phone. 

4.  The  written  questionnaire  was  sent  by  mail  to  parents  of  students  in  each  school.  In 
addition,  the  high  school  students  completed  the  questionnaire  in  class,  while  the 
teachers  completed  their  questionnaire  at  the  school  during  a common  meeting  session. 

5.  After  the  written  questionnaires  were  completed,  a telephone  survey  was  conducted 
with  a random  sample  of  people  from  each  stakeholder  group.  Lists  provided  by  each 
school  were  used  to  conduct  the  required  number  of  telephone  interviews. 

7.  In  all  locations,  100-120  telephone  interviews  were  completed  with  parents  and 
students,  and  between  12  and  25  teachers  completed  the  telephone  survey.  The 
following  chart  identifies  the  overall  number  of  completed  interviews  for  each  location 
and  group. 


Number  of  Completed  Interviews 


Location 

Segment 

Parents 

Students 

Teachers 

Written 

Phone 

Written 

Phone 

Written 

Phone 

School  1 

N=18 

N=100 

N=66 

N=100 

N=20 

N=14 

School  2 

N=128 

N=100 

N=178 

N=120 

N=32 

N=12 

School  3 

N=62 

N=100 

N=161 

N=101 

N=27 

N=25 

School  4 

N=279 

N=103 

N=414 

N=102 

N=30 

N=23 

Summary  and  Conclusions 

l.  There  is  a high  degree  of  consistency  between  the  results  of  the  written  and  local 
telephone  surveys  for  teachers  and  parents  but  there  are  significant  differences  for 
students.  In  those  cases  where  there  is  a difference,  the  written  surveys  have 
consistently  lower  ratings. 

II.  Overall,  where  differences  occurred  between  the  written  and  phone  responses,  the 
written  responses  tended  to  be  lower  - even  for  parents. 

m.  The  provincial  and  local  telephone  surveys  showed  higher  levels  of  consistency  in  the 
results.  This  indicates  that  the  phone  methodology  used  in  both  surveys  tends  to  give 
similar  types  of  results. 
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IV.  Comparisons  were  made  between  two  groups  of  respondents  - those  who  completed  the 
phone  survey  and  who  also  had  returned  a written  questionnaire  versus  those  that 
completed  a phone  survey  but  did  not  return  a written  questionnaire.  There  were  no 
significant  differences  between  the  responses  for  these  two  groups. 

V.  The  school  jurisdictions  having  the  most  experience  in  administering  satisfaction 
surveys  showed  less  variance  in  survey  results.  This  suggests  that  the  training  and 
experience  needed  to  conduct  the  satisfaction  surveys  (whether  written  or  phone)  are 
critical  in  getting  accurate  and  consistent  results,  and  that  all  schools  and  jurisdictions 
would  need  a similar  approach  to  the  design  and  execution  of  the  surveys  if 
comparisons  between  jurisdictions  are  to  be  made. 

VI.  An  analysis  of  the  resources  necessary  to  administer  telephone  versus  mail  surveys  was 
conducted.  Overall,  the  tasks  required  for  mail  versus  telephone  questionnaires  are  very 
similar  except  that  mail  questionnaires  require  more  effort  to  prepare  the  mailout 
packages  and  they  are  sent  out  to  alL  parents.  In  contrast,  the  telephone  interview 
requires  more  effort  in  hiring,  training  and  supervising  personnel  to  conduct  the 
interviews. 

The  time  and  costs  involved  for  conducting  the  tasks  outlined  for  mail/  written  surveys 
versus  telephone  surveys  are  listed  below: 

■ the  mail/  written  surveys  included  those  mailed  out  to  parents  as  well  as  those 
completed  in  the  school  by  students  and  teachers. 

■ the  costs  for  the  schools  were  developed  by  taking  the  average  of  the  hourly 
rates  they  paid  personnel  to  conduct  various  tasks  (These  varied  from  $12.50 
to  $17.00  to  $29.00  per  hour  - an  average  of  $19.50  was  used). 

■ the  data  entry  and  data  analysis  costs  were  not  directly  incurred  by  the  schools 
but  reflect  the  costs  they  would  incur  if  they  performed  these  tasks  or  had  an 
outside  contractor  do  them. 

■ the  costs  to  conduct  these  tasks  also  have  been  listed  for  comparison  purposes. 

■ the  costs  do  not  include  consultation  time  but  strictly  refer  to  fieldwork,  data 
entry,  and  preparation  of  the  data  tables. 
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The  Government  of  Alberta  has  introduced  the  concept  of  three  year 
business  plans  for  its  departments,  agencies,  boards  and  commissions  as 
part  of  its  strategy  to  improve  public  services  within  budget  targets. 
Alberta  Education's  plan,  Meeting  the  Challenge  II  -Three- Year  Business 
Plan  1995/96  to  1997/98,  provides  direction  for  the  future  of  education. 
The  plan  calls  for  better  reporting  of  educational  information  to  enhance 
accountability  to  the  public  for  results  achieved.  To  this  end,  Alberta 
Education  commissioned  a research  study  to  collect  information  on  the 
satisfaction  levels  of  various  stakeholders  with  the  education  system. 
This  research  involved  3611  telephone  interviews  with  seven  different 
stakeholder  groups  between  March  8 and  April  7,  1995. 

In  addition  to  this  provincial  research,  it  is  anticipated  that  similar  types 
of  satisfaction  surveys  will  be  conducted  by  individual  schools  and 
school  jurisdictions  across  the  province  in  the  coming  years. 
Consequently,  it  was  decided  to  conduct  a pilot  project  that  would  have 
the  following  objectives: 

1.  To  compare  the  results  of  the  provincial  telephone  survey  with 
survey  results  at  the  school  level. 

2.  To  compare  the  results  from  telephone  interviews  with  the  results 
from  self  administered  questionnaires  (the  methodology  most 
likely  to  be  used  by  schools  and  school  jurisdictions). 

3.  To  assess  local  training  requirements  that  schools/school 
jurisdictions  have  for  conducting  self  administered  and/or 
telephone  interviews,  and  to  prepare  a training  manual  that  would 
assist  schools  in  training  local  staff  to  conduct  this  type  of 
research. 
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Four  school  jurisdictions  agreed  to  participate  in  the  pilot  project:  Rocky 

View  School  Division,  Grasslands  Regional  Division,  Golden  Hills 

Regional  Division,  and  Fort  McMurrary  Catholic  School  District. 

Representatives  of  each  of  the  jurisdictions  attended  a series  of  meetings 

to: 

1.  Identify  which  school  in  the  jurisdiction  would  be  the  site  for  the 
research. 

2.  Review  the  provincial  questions  that  would  be  asked  in  the 
surveys. 

3.  Develop  other  questions  that  would  be  included  in  the  survey 
that  reflected  specific  issues  that  the  school  or  jurisdiction 
wanted  to  probe. 

4.  Prepare  a written  questionnaire  that  would  be  sent  out  to  parents, 
students  and  teachers  in  this  school,  and  coordinating  the 
distribution  and  receipt  of  these  questionnaires. 

5.  Provide  lists  of  parents,  students  and  teachers  that  would  be  used 
to  conduct  telephone  interviews. 

The  specific  research  methodology  for  the  pilot  project  was  as  follows: 

1.  For  each  school  jurisdiction,  one  high  school  was  selected  to 
participate  in  the  study.  These  included  School  1,  School  2, 
School  3,  and  School  4. 

2.  School  staff  and  system  administration  prepared  a questionnaire 
for  each  school  that  included  some  key  questions  that  were  asked 
on  the  provincial  survey.  In  addition,  each  school/  school 
jurisdiction  included  a list  of  questions  that  probed  issues  on 
which  more  information  was  wanted,  but  were  not  part  of  the 
provincial  survey.  Three  separate  questionnaires  were  prepared 
for  each  school/  school  jurisdiction:  one  for  parents,  one  for  high 
school  students,  and  one  for  teachers. 

3.  Each  school  jurisdiction  prepared  a written/  self-completion 
questionnaire,  and  one  that  could  be  administered  over  the  phone. 

4.  The  written  questionnaire  was  sent  by  mail  to  parents  of  students 
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in  each  school.  In  addition,  the  high  school  students  completed 
the  questionnaire  in  class,  while  the  teachers  completed  their 
questionnaire  at  the  school  during  a common  meeting  session. 

5.  Written  questionnaires  were  completed  during  the  last  two  weeks 
of  June. 

6.  After  the  written  questionnaires  were  completed,  a telephone 
survey  was  conducted  with  a random  sample  of  people  from  each 
stakeholder  group.  These  interviews  were  conducted  during  the 
first  three  weeks  of  July.  Lists  provided  by  each  school  were 
used  to  conduct  the  required  number  of  telephone  interviews. 

7.  Of  the  parents  and  students  who  completed  the  telephone  survey, 
some  had  completed  the  written  questionnaire  and  some  had  not. 
Originally  half  these  interviews  were  to  be  conducted  prior  to  the 
written  questionnaire  being  sent  out  and  half  were  to  be 
conducted  after.  However  due  to  time  constraints  it  was  agreed 
by  Alberta  Education  to  conduct  all  telephone  interviews  after 
the  written  questionnaires  had  been  returned. 

8.  In  all  locations,  100-120  telephone  interviews  were  completed 
with  parents  and  students,  and  between  12  and  25  teachers 
completed  the  telephone  survey.  The  following  chart  identifies 
the  overall  number  of  completed  interviews  for  each  location  and 
group. 


Number  of  Completed  Interviews 


Location 

Segment  j 

Parents 

Students 

Teachers 

Written 

Phone 

Written 

Phone 

Written 

Phone 

School  1 

N=18 

N=100 

N=66 

N=100 

N=20 

N=14 

School  2 

N=128 

N=100 

N=178 

N=120 

N=32 

N=12 

School  3 

N=62 

N=100 

N=161 

N=101 

N=27 

N=25 

School  4 

N=279 

N=103 

N=414 

N=102 

N=30 

N=23 

9.  All  telephone  interviews  were  conducted  by  professional 
interviewers  from  Environics  West  using  central  location  CATI 
(computer  assisted  telephone  interviewing)  facilities  in  Calgary. 


10.  A detailed  list  of  the  completion  rates  for  telephone  and  written 
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questionnaires  appear  in  the  appendix.  It  should  be  noted  that 
the  completion  rates  for  the  telephone  surveys  were  the  same  or 
higher  than  those  for  the  provincial  phone  survey. 

The  following  chart  compares  the  completion  and  response  rates  for  the 
written  and  phone  surveys. 


Overall  Comparison  of  Completed/Response  Rates 


Location 

Segment 

Parents 

Students 

Teachers 

Written 

Phone 

Written 

Phone 

Written 

Phone 

School  1 

13% 

85% 

35% 

92% 

91% 

88% 

School  2* 

18% 

91% 

94% 

73% 

78% 

92% 

School  3 

22% 

83% 

57% 

89% 

90% 

96% 

School  4 

56% 

86% 

83% 

88% 

91% 

96% 

* Please  note  that  School  2 is  an  estimate  since  only  a random  selection 

(and  not  all)  students  completed  the  survey. 

In  terms  of  analysis,  the  following  were  conducted: 

1.  All  written  questionnaires  that  were  returned  to  the  schools  were 
forwarded  to  Environics  West  where  professional  coders  input 
the  data  into  computer  files. 

2.  The  common  provincial  questions  were  isolated  in  both  the 
written  and  telephone  surveys  and  put  into  tables  (see  the 
following  section)  that  outlined  the  response  for  each  question. 
These  tables  break  down  the  result  for  each  question  by  school 
jurisdiction  and  stakeholder  group.  In  addition  the  results  are 
broken  down  by  methodology  (i.e.,  the  provincial  telephone 
survey,  the  written  questionnaire,  and  the  telephone  survey 
conducted  at  each  school). 

3.  Based  on  these  results,  a "SigTest"  was  calculated  for  difference 
in  each  school  jurisdiction  and  stakeholder  group  to  determine  if 
the  results  obtained  from  each  methodology  were  statistically 
significant  or  not.  Significantly  different  results  (at  the  95% 
level)  are  indicated  in  the  tables  according  to  the  following  code: 
1 = the  results  between  the  provincial  survey  and  the  written 
questionnaire  are  significantly  different,  2 = the  results  between 
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the  provincial  survey  and  the  local  telephone  survey  are 
significantly  different,  and  3 = the  results  between  the  written 
questionnaire  and  the  telephone  survey  are  significantly  different. 
If  there  was  no  significant  difference,  a " - " appears.  It  should 
be  noted  that  the  tests  were  conducted  on  the  "top  two"  box 
scores  for  all  questions  that  used  satisfaction  or  agreement  scales 
(i.e.,  very  satisfied  and  satisfied  scores  were  combined,  or 
strongly  agree  and  agree  scores  were  combined).  (What  is  a 
Sigtest?  T-Test  or  Student's  T is  designed  for  comparing  the 
difference  between  two  means  by  comparing  means  between  two 
subsamples,  T-Test  determines  if  the  difference  is  simply  due  to 
chance  or  indicates  a true  difference  between  the  two  samples.) 

4.  The  confidence  intervals  (at  the  95%  level)  for  the  sample  sizes 
obtained  in  each  stakeholder  can  be  determined  as  follows: 


N= 

25 

± 19.6% 

50 

± 13.9% 

100 

± 9.8% 

150 

± 8.2% 

200 

± 7.1% 

300 

± 5.8% 

400 

± 5.0% 

(i.e.,  with  a sample  size  of  400  respondents  the  results  of  the 
survey  will  be  reliable  to  within  +/-  5%  at  the  95%  confidence 
level.) 

5.  In  addition  to  this  analysis,  computer  tables  were  prepared  for 
each  school  jurisdiction  that  outlined  all  the  responses  for  both 
the  provincial  questions  as  well  as  their  own  specific  questions. 
Each  school  jurisdiction  received  only  its  own  results.  Alberta 
Education  received  copies  of  all  results. 
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The  research  results  are  broken  down  by  each  stakeholder  group  - 
students,  teachers  and  parents  and  an  analysis  are  presented  for  each.  A 
copy  of  the  questions  asked  of  each  group  appear  in  the  appendix. 

4.1  Students 

4.1.1  Satisfaction  with  the  Quality  of  Education 

Table  1 outlines  the  results  from  the  survey  about  student 
satisfaction  with  the  quality  of  education  that  they  are  receiving. 
In  all  cases,  the  results  between  the  provincial  telephone  survey 
and  the  written  survey  were  significantly  different.  The  written 
survey  results  ranged  from  5%  to  24%  lower  than  the  provincial 
survey.  However,  with  the  exception  of  School  1,  the  results 
between  the  provincial  survey  and  the  local  telephone  survey 
were  not  significantly  different.  When  comparing  the  written 
survey  and  the  local  telephone  survey  the  results  in  all  cases 
were  significantly  different.  The  written  results  ranged  from  7% 
to  16%  lower. 

4.1.2  Expectations  and  Student  Opportunities  for 
Success 

Tables  2 to  7 include  questions  relating  to  the  expectations  of 
students  and  the  opportunities  they  have  to  influence  the  way 
they  learn.  Overall,  there  is  much  more  consistency  in  the  results 
between  the  provincial,  and  the  locally  written  and  telephone 
surveys.  The  extreme  is  School  2,  where  there  were  no 
significant  differences  between  the  results  of  the  surveys  on  any 
questions.  The  questions  that  tended  to  show  the  greatest 
differences  focused  on  teachers  using  a variety  of  approaches, 
teachers  providing  help  and  support,  and  students  being 
challenged  to  do  their  best.  In  all  cases  the  written  results  were 
lower  than  the  telephone  survey  by  15%  to  20%  on  average. 

4.1.3  Improvement  on  Student  Skills 

Tables  8 to  25  include  questions  that  relate  to  students' 
perceptions  about  how  much  high  school  is  helping  them 
improve  in  18  different  skill  areas.  For  each  area  they  were  asked 
whether  high  school  was  helping  them  improve  a lot,  some  or  a 
little.  In  reviewing  these  results,  there  is  a high  degree  of 
consistency  between  the  written  and  telephone  surveys.  In 
looking  at  where  differences  do  exist,  the  results  in  School  4 tend 
to  show  more  differences  while  the  other  locations  show  a very 
high  degree  of  consistency.  Once  again,  where  differences  do 
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exist,  the  written  results  are  lower  than  the  telephone  survey  by 
about  15%  to  18%.  It  should  be  noted  however,  that  these 
differences  tend  to  be  focused  on  the  upper  end  of  the  scale;  that 
is,  fewer  students  in  the  written  survey  agreed  that  high  school 
was  helping  them  improve  "a  lot",  and  more  of  these  students 
were  likely  to  say  it  was  helping  them  "some". 

The  biggest  differences  were  between  the  provincial  survey 
results  and  the  written  survey.  Once  again,  the  written  results 
were  lower  than  the  provincial  results  by  7%  to  28%.  These 
differences  too  were  concentrated  on  the  upper  end  of  the  scale, 
where  the  written  responses  showed  fewer  students  indicating 
that  high  school  helped  them  "a  lot"  in  certain  areas. 

4.1.4  Behavioral  Expectations 

Table  26  and  27  include  questions  relating  to  behavioral 
expectations  and  the  degree  to  which  students  "follow  the  rules". 
Again,  there  is  a high  degree  of  consistency  between  the  written 
and  telephone  results  with  the  exception  of  School  4,  where  the 
written  results  are  lower.  It  should  be  noted  that  part  of  the 
reason  that  written  results  are  lower  is  the  higher  number  of 
students  in  the  written  surveys  that  gave  a "don't  know"  or  "no 
response"  for  the  question.  This  comment  applies  to  all  written 
results. 

In  all  but  one  case,  the  provincial  results  are  significantly 
different  from  the  written  results,  with  the  written  results  being 
lower  by  6%  to  26%. 

4.1.5  Opportunities  for  Involvement 

Tables  28  and  29  include  questions  relating  to  students' 
satisfaction  with  opportunities  to  be  involved  in  outside  activities 
and  in  decisions  that  effect  student  life.  These  tables  show 
significant  differences  between  the  written  and  telephone  surveys 
that  range  from  15%  to  28%.  Again,  part  of  this  difference  is  the 
number  of  "don't  know"  and  "no  response"  answers  in  the  written 
surveys.  Outside  of  School  1 on  one  question,  there  are  no 
significant  differences  between  the  provincial  survey  and  the 
telephone  survey.  However,  there  are  significant  differences 
between  the  provincial  survey  and  the  written  survey  in  all 
locations. 
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4.2  Parents 

4.2.1  Satisfaction  with  the  Quality  of  Education 

Tables  28  to  30  include  questions  about  parents'  satisfaction  with 
the  quality  of  education.  Overall,  there  is  high  degree  of 
consistency  between  the  written  and  phone  surveys.  The  lone 
exception  is  School  3,  where  23%  more  parents  were  satisfied  in 
the  phone  survey  as  compared  to  the  written  survey.  There  also 
is  also  a fairly  high  degree  of  consistency  between  the  provincial 
survey  and  the  written  and  phone  surveys. 

4.2.2  Information  Access  and  Quality 

Tables  33  and  34  contain  questions  about  information  access  and 
quality.  There  are  no  significant  differences  between  the  written 
and  phone  surveys.  There  are  some  significant  differences 
between  the  provincial  survey  and  the  written  and  phone  surveys, 
particularly  for  School  1 and  School  3. 

4.3  Teachers 

4.3.1  Teacher  Preparation  and  Authority 

Tables  35  to  37  contain  questions  about  teacher  preparation  and 
authority.  There  are  no  significant  differences  between  the  results 
from  the  provincial,  written,  and  phone  surveys. 
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A number  of  conclusions  can  be  drawn  from  an  examination  of  the  results  of  the  surveys  as 
well  as  the  process  used  to  conduct  the  surveys: 

I.  There  is  a high  degree  of  consistency  between  the  results  of  the  written  and  local 
telephone  surveys  for  teachers  and  parents  but  there  are  significant  differences  for 
students.  In  those  cases  where  there  is  a difference,  the  written  surveys  have 
consistently  lower  ratings.  The  methodological  reasons  for  this  could  be  as  follows: 

a.  Students  completed  the  questionnaires  in  the  classroom  under  the  supervision 
of  teachers.  Although  students  completed  their  own  questionnaires,  there  may 
be  some  "peer  effect"  where  students  are  less  likely  to  give  higher  scores  (i.e., 
use  the  upper  end  of  the  scales).  This  seems  to  be  reinforced  by  the  fact  they 
give  teachers  lower  ratings  and  tend  to  use  the  "some"  rating  versus  the  "a  lot" 
rating  when  evaluating  areas  of  improvement. 

b.  There  was  more  time  pressure  for  students  to  complete  the  survey  in  a 
classroom  setting  and  there  was  no  real  opportunity  to  ask  questions  for 
clarification.  Over  the  phone,  the  students  were  taken  through  the  survey  step 
by  step  and  had  the  opportunity  to  ask  questions  if  they  needed  help.  In  the 
classroom  there  may  have  been  a tendency  to  rush  through  the  survey  and  not 
give  questions  adequate  thought. 

c.  There  were  more  "don't  know"  and  "no  responses"  on  the  written  questionnaires 
as  compared  to  the  phone  surveys.  In  some  cases  the  differences  can  be 
explained  partially  by  the  higher  percentage  of  students  who  did  not  answer 
specific  questions  on  the  written  survey.  This  could  occur  because  they  were 
not  encouraged  to  respond  (as  in  the  telephone  survey)  to  some  questions.  This 
was  particularly  apparent  in  the  "behavioral  expectations"  and  "opportunities  for 
involvement"  questions. 

d.  The  student  written  questionnaires  were  much  longer  than  the  parent  or  teacher 
questionnaires,  and  this  may  have  resulted  in  some  fatigue;  this  could  have 
contributed  to  some  the  "don't  know"  and  "no  response". 

e.  The  completion  rates  for  students  ranged  from  24%  to  83%.  This  could  account 
for  some  variability  in  the  results  (i.e.,  there  could  be  some  response 
bias/differences  because  large  numbers  of  students  in  some  schools  did  not 
complete  the  written  survey.  Those  students  who  completed  the  survey  may 
have  been  more  negative  in  their  responses  than  students  who  did  not  complete 
the  survey). 
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E.  Overall,  where  differences  occurred  between  the  written  and  phone  responses,  the 
written  responses  tended  to  be  lower  - even  for  parents.  While  possible  reasons  for  the 
students'  lower  ratings  have  been  cited,  for  parents  this  may  be  due  to  the  types  of 
people  responding  to  the  survey  (i.e.,  there  is  a tendency  in  written  satisfaction  surveys 
to  get  more  negative  responses  because  those  people  are  more  likely  to  send  back  their 
questionnaires). 

m.  The  provincial  and  local  telephone  surveys  showed  higher  levels  of  consistency  in  the 
results.  This  indicates  that  the  phone  methodology  used  in  both  surveys  tends  to  give 
similar  types  of  results.  Where  differences  did  exist  (with  local  surveys  showing  lower 
ratings)  there  are  a number  of  explanations: 

a.  These  reflect  true  differences  between  the  regions  and  the  provincial  survey. 
This  is  supported  by  the  fact  that  for  many  questions  there  is  n£l  significant 
differences  across  all  locations.  That  is,  some  locations  show  lower  ratings 
while  others  do  not. 

b.  The  surveys  were  conducted  at  different  times  of  the  year.  All  stakeholder 
groups  completed  the  written  survey  at  the  £nd  of  the  school  year.  This  would 
have  the  greatest  effect  for  students  who  would  have  had  a more  precise  idea 
of  their  grades  and  who  had  a full  year  to  evaluate,  rather  than  nine  months  of 
the  school  year  as  in  the  provincial  survey. 

c.  The  sample  sizes  of  the  provincial  studies  were  much  larger  than  those  in  the 
pilot  project.  That  is,  the  confidence  interval  for  the  pilot  project  surveys  would 
be  larger  because  of  the  smaller  sample  sizes  used,  and  thus,  more  variation 
would  be  expected  in  the  results  whether  these  are  significantly  different  or  not. 

IV.  Comparisons  were  made  between  two  groups  of  respondents  - those  who  completed  the 
phone  survey  and  who  had  also  returned  a written  questionnaire  versus  those  that 
completed  a phone  survey  but  did  not  return  a written  questionnaire.  There  were  no 
significant  differences  between  the  responses  for  these  two  groups.  This  result  is 
interesting  given  the  earlier  finding  that  in  some  cases  (particularly  for  students)  the 
written  questionnaire  produced  lower  ratings.  This  could  have  occurred  for  a number 
of  reasons: 

a.  People  are  more  likely  to  give  negative  ratings  in  a written  questionnaire  than 
a personal  interview;  that  is,  people  were  more  reluctant  to  give  lower  ratings 
over  the  phone. 

b.  With  students  (who  completed  the  interview  in  class),  the  phone  interview 
removed  the  "peer"  effect.  That  is,  it  allowed  them  to  give  their  opinion 
without  feeling  direct  or  indirect  pressure  from  others. 
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c.  Over  the  phone,  the  number  of  "don't  knows"  and  "no  responses"  is  drastically 
reduced  (compared  to  written  questionnaires).  This  would  mean  that  more  of 
the  people  who  made  "don't  know/  no  responses"  in  the  written  questionnaires 
in  fact  gave  an  opinion  on  these  questions  in  the  phone  survey,  and  their  ratings 
tended  to  be  somewhat  higher. 

V.  The  school  jurisdictions  (particularly  that  for  School  2)  having  the  most  experience  in 
administering  satisfaction  surveys  showed  less  variance  in  survey  results.  This  suggests 
that  the  training  and  experience  needed  to  conduct  the  satisfaction  surveys  (whether 
written  or  phone)  are  critical  in  getting  accurate  and  consistent  results,  and  that  all 
schools  and  jurisdictions  would  need  a similar  approach  to  the  design  and  execution  of 
the  surveys  if  comparisons  between  jurisdictions  are  to  be  made. 

VI.  The  choice  of  research  method  is  dependent  on  the  real  and  perceived  level  of  effort 
and  resources  required  to  conduct  the  research.  It  was  clear  in  the  early  discussions 
with  each  school  jurisdiction  that  there  was  significant  resistance  to  conducting 
satisfaction  surveys  (and  particularly  those  by  phone)  because  of  resource  and  timing 
constraints.  This  leads  to  a number  of  observations. 

a.  Timing 

Satisfaction  surveys  are  seen  as  viable  only  if  they  are  conducted  when  staff  has 
"down  time".  In  this  case,  it  means  the  surveys  would  be  conducted  during  the 
last  two  weeks  of  June. 

b.  Data  Analysis 

There  was  no  consistency  between  schools  about  how  they  would  input  the 
survey  results  and  report  the  results.  School  2 was  the  most  sophisticated  and 
experienced  of  all  groups  in  this  area.  This  suggests  that  school  jurisdictions 
will  need  some  support  in  selecting  a simple  way  to  input  and  analyze  the  data 
either  internally  (through  common  software)  or  externally  (as  School  2 does  by 
having  an  outside  company  input  and  analyze  the  data). 

c.  Modeling/  Mentoring 

As  mentioned  earlier,  School  2 had  the  most  sophisticated  and  well-developed 
measurement  system  of  all  school  jurisdictions  in  this  pilot.  It  also  reported  the 
results  in  the  most  "actionable"  form.  It  would  be  useful  to  use  School  2’s 
measurement  system  as  a "model"  for  other  school  jurisdictions  and  see  if  some 
formal  approach  could  be  instituted  to  have  other  schools  view  first  hand  School 
2's  approach. 
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d.  Staffing 

Jurisdictions  did  not  want  to  hire  additional  staff:  They  wanted  to  use  existing 
staff  to  administer  the  questionnaires.  This  suggests  that  training  each  year  will 
be  critical  since  there  may  be  changes  in  the  people  who  are  administering  the 
survey  in  the  school,  or  if  the  same  people  are  administrating  the  survey  they 
may  have  "forgotten"  the  exact  procedures  to  be  used.  This  latter  point 
emphasizes  the  need  for  training  procedures/manuals  and  protocols. 

e.  Resources  and  Tasks  Needed  for  Mail  versus  Phone  Surveys 

The  prime  objections  to  using  phone  surveys  were  the  need  for  people  to 
conduct  the  surveys  and  the  time  required.  A chart  has  been  included  in  the 
appendix  to  highlight  the  amount  of  time  required  to  complete  and  enter  the 
data  from  the  surveys.  The  biggest  benefits  of  conducting  phone  surveys  would 
be  the  removal  of  respondent  bias  in  the  written  surveys  and  the  speed  with 
which  the  surveys  could  be  conducted.  The  biggest  drawbacks  would  be  the 
need  to  train  staff  or  volunteers  so  that  they  don't  bias  the  way  questions  are 
asked. 

In  terms  of  resources  and  tasks,  the  following  table  identifies  some  of  the  key  tasks  and 
resources  required  in  doing  mail  and  telephone  surveys. 


COMPARISON  OF  TASKS  FOR 
MAIL  VERSUS  TELEPHONE  INTERVIEWS 

TASK 

MAIL 

TELEPHONE 

Questionnaire 

Design 

Identify  key  question  areas  and 
prepare  questionnaire. 

Prepare  introduction  letter 
Prepare  instructions  on  the  survey 
- these  must  be  very  clear  so 
respondents  complete  the  survey 
properly. 

Pre-test  questionnaire. 

Photocopy  questionnaire  for 
sample. 

Prepare  mailout  package. 

Identify  key  question  areas  and 
prepare  questionnaire.  I 

Pre-test  questionnaire. 

Photocopy  questionnaires  for 
interviewers. 

Sample 

Assemble  all  available  mail 
listings  for  parents  and  verify 
address  information  for  accuracy. 
Distribute  survey  to  all  parents. 

Assemble  all  available  telephone 
listings  for  parents,  students  and 
teachers. 

Select  a random  sampling  of  | 

respondents  for  each  target 
group. 
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Supervision 

Supervision  is  required  for 
distribution,  data  entry  of 
completed  surveys  and 
verification. 

Recruit  and  train  phone 
interviewers. 

Supervision  is  required  for 
interviewing,  monitoring,  and 
verification  of  phone  interviews 
as  well  as  data  entry. 

Follow  Up 

Possible  requirement  of 
distributing  follow  up  letters  or 
making  follow  up  calls  to 
increase  response  rate. 

No  additional  follow  up  required. 

Data  Entry 

Data  enter  all  surveys  to 
spreadsheet  program. 

Code  all  open  end  responses 
(Training  required). 

Depending  on  response  rate,  the 
number  of  questionnaires  entered 
could  be  higher  than  for  a phone 
survey. 

Data  enter  all  surveys  to 
spreadsheet  program. 

Code  all  open  end  responses 
(Training  required). 

Data  Analysis  and 
Report  Presentation 

Statistical  program  required  for 
computer  analysis. 

Graphics  program  required  for 
report  preparation. 

Statistical  program  required  for 
computer  analysis. 

Graphics  program  required  for 
report  preparation. 

An  analysis  of  the  resources  necessary  to  administer  telephone  versus  mail  surveys  was 
conducted.  Overall,  the  tasks  required  for  mail  versus  telephone  questionnaires  are  very  similar 
except  that  mail  questionnaires  require  more  effort  to  prepare  the  mailout  packages  and  they 
are  sent  out  to  all  parents.  In  contrast,  the  telephone  interview  requires  more  effort  in  hiring, 
training  and  supervising  personnel  to  conduct  the  interviews. 

The  time  and  costs  involved  for  conducting  the  tasks  outlined  for  mail/  written  surveys  versus 
telephone  surveys  are  listed  below.  The  mail/  written  surveys  included  those  mailed  out  to 
parents  as  well  as  those  completed  in  the  school  by  students  and  teachers.  The  costs  for  the 
schools  were  developed  by  taking  the  average  of  the  hourly  rates  they  paid  personnel  to 
conduct  various  tasks  (These  varied  from  $12.50  to  $17.00  to  $29.00  per  hour  - an  average  of 
$19.50  was  used).  The  data  entry  and  data  analysis  costs  were  not  directly  incurred  by  the 
schools  but  reflect  the  costs  they  would  incur  if  they  performed  these  tasks  or  had  an  outside 
contractor  do  them.  The  costs  for  Environics  to  conduct  these  tasks  also  has  been  listed  for 
comparison  purposes.  The  Environics  costs  do  not  include  consultation  time  but  strictly  refer 
to  fieldwork,  data  entry,  and  preparation  of  the  data  tables.  It  should  be  noted  that  this  table 
refers  to  the  total  costs  for  conducting  the  interviews  for  all  four  schools  and  for  all  three 
stakeholder  groups.  Consequently,  all  "per  school"  costs  were  multiplied  by  four  to  get  total 
costs  for  a task. 
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In  reviewing  the  costs,  it  can  be  seen  that  written/  mail  surveys  are  approximately  half  those 
of  phone  surveys.  The  biggest  difference  occurs  in  briefing,  interviewing  and  supervision  of 
the  people  to  conduct  the  phone  surveys.  On  a per  school  basis  the  difference  works  out  to 
approximately  S2000.00. 
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NOTE:  Estimates  do  not  include  initial  interviewer,  supervisor  and  data  entry  (of  self  completes)  training  at  the  school  board  level.  It  is  assumed  that 
Environics  West  would  be  asked  for  assistance  regarding  coding  of  open  ended  responses  and  the  creation  of  data  frequency  tables. 
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= Difference  between  the  provincial  and  the  written  survey. 

= Difference  between  the  provincial  survey  and  the  phone  survey. 
= Difference  between  the  written  and  phone  survey. 


Environics  West 


Table  3 

School  4 

SigTest 

' 

• 

i 

Phone 
N = 102 

oo 

m 

o 

Written 

N=414 

m 

OO 

- 

School  3 

SigTest 

1,3 

• 

• 

Phone 
N = 101 

06 

oo 

CN 

Written 

N=161 

74 

24 

CN 

e 

u 

School  2 

SigTest 

• 

• 

• 

Phone 

N=120 

r- 

oo 

o 

Written 
N = 178 

78 

20 

CN 

Students  : Q.4  I am  clear  as  to  what  I am  expected  to  lea 

School  1 

SigTest 

• 

i { 

1 

Phone 
N = 100 

94 

VO 

O 

W ritten 
N=66 

86 

o 

Provincial 

N=800 

r- 

00 

CO 

o 

Strongly  Agree/ 
Agree 

Disagree 

Strongly  Disagree 

Don’t  Know/No  Response 

Table  4 

School  4 

SigTest 

1,2,3 

• 

• 

Phone 

N=102 

06 

o 

o 

Written 

N=414 

72 

23 

School  3 

SigTest 

- 

• 

t 

Phone 
N = 101 

75 

24 

- 

Written 

N=161 

69 

30 

- 

School  2 

SigTest 

I i ; 

i 

■ 

Phone 
N= 1 20 

83 

VO 

CN 

Written 

N=178 

76 

22 

- 

Students  : Q.4  I am  being  challenged  to  do  my  best. 

School  1 

SigTest 

- 

I 

1 

Phone 

N=100 

78 

CN 

- 

Written 

N=66 

69 

30 

CN 

Provincial 

N=800 

00 

Ov 

o 

Strongly  Agree/ 
Agree 

Disagree 

Strongly  Disagree 

Don’t  Know/No  Response 

T5 

>» 


00 

c/5 


= Difference  between  the  provincial  and  the  written  survey. 

= Difference  between  the  provincial  survey  and  the  phone  survey. 
= Difference  between  the  written  and  phone  survey. 
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= Difference  between  the  provincial  and  the  written  survey. 

= Difference  between  the  provincial  survey  and  the  phone  survey. 
= Difference  between  the  written  and  phone  survey. 
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= Difference  between  the  provincial  and  the  written  survey. 

= Difference  between  the  provincial  survey  and  the  phone  survey. 
= Difference  between  the  written  and  phone  survey. 
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= Difference  between  the  provincial  and  the  written  survey. 

= Difference  between  the  provincial  survey  and  the  phone  survey. 
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= Difference  between  the  provincial  and  the  written  survey. 

= Difference  between  the  provincial  survey  and  the  phone  survey. 
= Difference  between  the  written  and  phone  survey. 
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Appendix  A:  Response/Completion  Rates  for 

Telephone  and  Written  Questionnaires 


Environics  West 


A.1  Response  Rates  for  Self  Administered  Survey 

The  chart  below  identifies  the  response  rates  for  the  written  questionnaires  that  were 
completed  for  each  group.  Column  1 represents  the  total  population  for  that  group  as 
determined  by  school  lists.  Column  2 is  the  number  of  self-completed  surveys  that  were 
completed  at  school  or  returned  through  the  mail  that  the  school  forwarded  to  Environics 
West.  Column  3 is  the  response  rate  based  on  dividing  column  2 by  column  3. 


Response  Rate  - Self  Administered  Survey 


Location 

Percentage  Returned 

Parents 

Students 

Teachers 

l 

2 

3 

l 

2 

3 

l 

2 

3 

School  1 

142 

18 

13% 

188 

66 

35% 

22 

20 

91% 

School  2 

729 

128 

18% 

190 

178 

94% 

41 

32 

78% 

School  3 

284 

62 

22% 

284 

161 

57% 

30 

27 

90% 

School  4 

501 

279 

56% 

501 

414 

83% 

33 

30 

91% 

Note:  The  response  rates  are  based  upon  the  assumption  that  100%  of  each  target 
population  received  a self-administered  survey.  Consequently,  the  following  were  not 
considered  when  calculating  the  response  rates: 

Percentage  of  mailed  questionnaires  returned  due  to  incorrect  address/  moved,  etc.; 
Percentage  of  students/teachers  who  were  absent  the  day  the  self  administered  survey 
was  distributed; 

Percentage  of  students/teachers  who  were  present,  but  did  not  complete  the  self 
administered  survey  for  whatever  reason; 

Percentage  of  senior  students  who  did  not  have  a class  during  time  of  survey 
distribution. 

Percentage  of  students  who  may  have  had  a field  trip  the  day  of  survey  distribution. 
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The  following  chart  compares  the  completion  and  response  rates  for  the  written  and  phone 
surveys. 


Overall  Comparison  of  Completed/Response  Rates 


Location 

Segment 

Parents 

Students 

Teachers 

Written 

Phone 

Written 

Phone 

Written 

Phone 

School  1 

13% 

85% 

35% 

92% 

91% 

88% 

School  2* 

18% 

91% 

94% 

73% 

78% 

92% 

School  3 

22% 

83% 

57% 

89% 

90% 

96%  | 

School  4 

56% 

86% 

83% 

88% 

91% 

96% 

* Please  note  that  School  2 is  an  estimate  since  only  a random  selection  (and  not  all) 
students  completed  the  survey. 

A.2  Call  Records,  Completion  Rates  and  Confidence  Ranges 

A table  has  been  included  for  each  stakeholder  group  outlining  the  details  of  the  calls  that 
were  made.  The  completion  rate  has  been  included  at  the  bottom  of  the  table  that  indicates 
what  percentage  of  people  contacted  agreed  to  complete  the  survey.  For  comparison 
purposes  with  the  provincial  survey,  the  completion  rates  for  parents  were  54%,  for  students 
76%,  and  for  teachers  79%. 
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COMPLETION  RATE  - PARENTS  (School  1)  - TELEPHONE  SURVEY 


■ TOTAL  NUMBER  OF  CALLS 

179 

100.00% 

■ Non-Valid/  Non-residential/  household  not  eligible1 

- Fax/Modem/cellular/pager  listings 

- Business  listings 

- Wrong  numbers 

- Duplicate  Numbers 

- Non  Qualified  respondents 

- Other 

12 

6.70% 

■ Numbers  not  in  Service 

8 

4.47% 

SUBTOTAL 

20 

11.17% 

■ NEW  BASE 

159 

88.83% 

■ No  Answer/Busy/Respondent  not  Available2 

- Answering  Machines 

- Voice  Mail 

- Call  Backs 

39 

21.79% 

■ Communication/Language  Barrier 

3 

1.68% 

■ Incomplete  Interviews 

- Respondent  Terminated 

- Quota  Filled  (gender,  geographic  location  etc. 

■ Refusals3 

17 

9.50% 

SUBTOTAL 

59 

32.96% 

NET  COMPLETIONS 

100 

55.87% 

COMPLETION  RATE: 

Completes  / New  Base  - (No  Answer  + Comm.  Lang.) 

85% 

A minimum  of  three  calls  were  made  to  a household  before  classifying  it  as  a "no 
answer". 

Refers  to  contacts  made  where  the  telephone  listing  or  respondent  did  not  represent 
a valid  member  of  the  population 

Refers  to  the  number  of  telephone  numbers  in  the  sample  that  could  not  be 
contacted  even  after  five  callbacks 

Refers  to  the  number  of  people  contacted  who  were  qualified  but  who  did  not  want 
to  participate  in  the  survey. 
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COMPLETION  RATE  - PARENTS  (School  2)  - TELEPHONE  SURVEY 


■ TOTAL  NUMBER  OF  CALLS 

373 

100.00% 

■ Non-valid/  non-residential/  household  not  eligible1 

- Fax/Modem/cellular/pager  listings 

- Business  listings 

- Wrong  numbers 

- Duplicate  Numbers 

- Non  Qualified  respondents 

- Other 

15 

4.02% 

■ Numbers  not  in  Service 

9 

2.41% 

SUBTOTAL 

24 

6.43% 

■ NEW  BASE 

349 

93.57% 

■ No  Answer/Busy/Respondent  not  Available2 

- Answering  Machines 

- Voice  Mail 

- Call  Backs 

239 

64.08% 

■ Communication/Language  Barrier 

- 

0.00 

■ Incomplete  Interviews 

- Respondent  Terminated 

- Quota  Filled  (gender,  geographic  location  etc. 

2 

0.54% 

■ Refusals3 

8 

2.14% 

SUBTOTAL 

249 

66.76% 

NET  COMPLETIONS 

100 

26.81% 

COMPLETION  RATE: 

Completes  / New  Base  - (No  Answer  + Comm.  Lang.) 

90.91% 

SAMPLE  NOT  ACCESSED  - 270 

A minimum  of  three  calls  were  made  to  a household  before  classifying  it  as  a "no 
answer". 

Refers  to  contacts  made  where  the  telephone  listing  or  respondent  did  not  represent 
a valid  member  of  the  population 

Refers  to  the  number  of  telephone  numbers  in  the  sample  that  could  not  be 
contacted  even  after  five  callbacks 

Refers  to  the  number  of  people  contacted  who  were  qualified  but  who  did  not  want 
to  participate  in  the  survey. 
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COMPLETION  RATE  PARENTS  (School  3)  - TELEPHONE  SURVEY 


■ TOTAL  NUMBER  OF  CALLS 

178 

100.00% 

■ Non-Valid/  Non-residential/  household  not  eligible1 

- Fax/Modem/cellular/pager  listings 

- Business  listings 

- Wrong  numbers 

- Duplicate  Numbers 

- Non  Qualified  respondents 

- Other 

26 

14.61% 

■ Numbers  not  in  Service 

7 

3.93% 

SUBTOTAL 

33 

18.54% 

■ NEW  BASE 

145 

81.46% 

■ No  Answer/Busy/Respondent  not  Available2 

- Answering  Machines 

- Voice  Mail 

- Call  Backs 

24 

13.48% 

■ Communication/Language  Barrier 

1 

0.56% 

■ Incomplete  Interviews 

- Respondent  Terminated 

- Quota  Filled  (gender,  geographic  location  etc 

.1 

0.56% 

■ Refusals3 

19 

10.67% 

SUBTOTAL 

45 

25.28% 

NET  COMPLETIONS 

100 

56.18% 

COMPLETION  RATE: 

Completes  / New  Base  - (No  Answer  + Comm.  Lang.) 

83.33% 

SAMPLE  NOT  ACCESSED  - 72 


A minimum  of  three  calls  were  made  to  a household  before  classifying  it  as  a "no 
answer". 

Refers  to  contacts  made  where  the  telephone  listing  or  respondent  did  not  represent 
a valid  member  of  the  population 

Refers  to  the  number  of  telephone  numbers  in  the  sample  that  could  not  be 
contacted  even  after  five  callbacks 

Refers  to  the  number  of  people  contacted  who  were  qualified  but  who  did  not  want 
to  participate  in  the  survey. 


Pilot  Project  50 

Alberta  Education 


Environics  West 


COMPLETION  RATE  - PARENTS  (School  4)  - TELEPHONE  SURVEY 


■ TOTAL  NUMBER  OF  CALLS 

160.00 

100.00% 

■ Non-Valid/  Non-residential/  household  not  eligible1 

- Fax/Modem/cellular/pager  listings 

- Business  listings 

- Wrong  numbers 

- Duplicate  Numbers 

- Non  Qualified  respondents 

- Other 

10 

6.25% 

■ Numbers  not  in  Service 

2 

1.25% 

SUBTOTAL 

12 

7.50% 

■ NEW  BASE 

148 

92.50% 

■ No  Answer/Busy/Respondent  not  Available2 

- Answering  Machines 

- Voice  Mail 

- Call  Backs 

26 

16.25% 

■ Communication/Language  Barrier 

2 

1.25% 

■ Incomplete  Interviews 

- Respondent  Terminated 

- Quota  Filled  (gender,  geographic  location  etc. 

■ Refusals3 

17 

10.63% 

SUBTOTAL 

45 

28.13% 

NET  COMPLETIONS 

103 

64.38% 

COMPLETION  RATE: 

Completes  / New  Base  - (No  Answer  + Comm.  Lang.) 

86% 

SAMPLE  NOT  ACCESSED  - 282 


A minimum  of  three  calls  were  made  to  a household  before  classifying  it  as  a "no 
answer". 

Refers  to  contacts  made  where  the  telephone  listing  or  respondent  did  not  represent 
a valid  member  of  the  population 

Refers  to  the  number  of  telephone  numbers  in  the  sample  that  could  not  be 
contacted  even  after  five  callbacks 

Refers  to  the  number  of  people  contacted  who  were  qualified  but  who  did  not  want 
to  participate  in  the  survey. 
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COMPLETION  RATE  - STUDENTS  (School  1)  - TELEPHONE  SURVEY 


■ TOTAL  NUMBER  OF  CALLS 

179 

100.00% 

■ Non-Valid/  Non-residential/  household  not  eligible1 

- Fax/Modem/cellular/pager  listings 

- Business  listings 

- Wrong  numbers 

- Duplicate  Numbers 

- Non  Qualified  respondents 

- Other 

6 

3.35% 

■ Numbers  not  in  Service 

8 

4.47% 

SUBTOTAL 

14 

7.82% 

■ NEW  BASE 

165 

92.18% 

■ No  Answer/Busy/Respondent  not  Available2 

- Answering  Machines 

- Voice  Mail 

- Call  Backs 

56 

31.28% 

■ Communication/Language  Barrier 

- 

■ Incomplete  Interviews 

- Respondent  Terminated 

- Quota  Filled  (gender,  geographic  location  etc. 

■ Refusals3 

9 

5.03% 

SUBTOTAL 

65 

36.31% 

NET  COMPLETIONS 

100 

55.87% 

COMPLETION  RATE: 

Completes  / New  Base  - (No  Answer  + Comm.  Lang.) 

91.74% 

A minimum  of  three  calls  were  made  to  a household  before  classifying  it  as  a "no 
answer". 

Refers  to  contacts  made  where  the  telephone  listing  or  respondent  did  not  represent 
a valid  member  of  the  population 

Refers  to  the  number  of  telephone  numbers  in  the  sample  that  could  not  be 
contacted  even  after  five  callbacks 

Refers  to  the  number  of  people  contacted  who  were  qualified  but  who  did  not  want 
to  participate  in  the  survey. 
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COMPLETION  RATE  - STUDENTS  (School  2)  - TELEPHONE  SURVEY 


■ TOTAL  NUMBER  OF  CALLS 

352 

100.00% 

■ Non-Valid/  Non-residential/  household  not  eligible1 

- Fax/Modem/cellular/pager  listings 

- Business  listings 

- Wrong  numbers 

- Duplicate  Numbers 

- Non  Qualified  respondents 

- Other 

16 

4.55% 

■ Numbers  not  in  Service 

7 

1.99% 

SUBTOTAL 

23 

6.53% 

■ NEW  BASE 

329 

93.47% 

■ No  Answer/Busy/Respondent  not  Available2 

- Answering  Machines 

- Voice  Mail 

- Call  Backs 

165 

46.88% 

■ Communication/Language  Barrier 

- 

■ Incomplete  Interviews 

- Respondent  Terminated 

- Quota  Filled  (gender,  geographic  location  etc. 

18 

5.11% 

■ Refusals3 

26 

7.39% 

SUBTOTAL 

209 

59.38% 

NET  COMPLETIONS 

120 

34.09% 

COMPLETION  RATE: 

Completes  / New  Base  - (No  Answer  + Comm.  Lang.) 

73.17% 

SAMPLE  NOT  ACCESSED  - 291 

A minimum  of  three  calls  were  made  to  a household  before  classifying  it  as  a "no 
answer". 

1 Refers  to  contacts  made  where  the  telephone  listing  or  respondent  did  not  represent 
a valid  member  of  the  population 

Refers  to  the  number  of  telephone  numbers  in  the  sample  that  could  not  be 
contacted  even  after  five  callbacks 

Refers  to  the  number  of  people  contacted  who  were  qualified  but  who  did  not  want 
to  participate  in  the  survey. 
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COMPLETION  RATE  STUDENTS  (School  3)  - TELEPHONE  SURVEY 


■ TOTAL  NUMBER  OF  CALLS 

208 

100.00% 

■ Non-Valid/  Non-residential/  household  not  eligible1 

- Fax/Modem/cellular/pager  listings 

- Business  listings 

- Wrong  numbers 

- Duplicate  Numbers 

- Non  Qualified  respondents 

- Other 

11 

5.29% 

■ Numbers  not  in  Service 

11 

5.29% 

SUBTOTAL 

22 

10.58% 

■ NEW  BASE 

186 

89.42% 

■ No  Answer/Busy/Respondent  not  Available2 

- Answering  Machines 

- Voice  Mail 

- Call  Backs 

72 

34.62% 

■ Communication/Language  Barrier 

■ Incomplete  Interviews 

- Respondent  Terminated 

- Quota  Filled  (gender,  geographic  location  etc. 

■ Refusals3 

13 

6.25% 

SUBTOTAL 

85 

40.87% 

NET  COMPLETIONS 

101 

48.56% 

COMPLETION  RATE: 

Completes  / New  Base  - (No  Answer  + Comm.  Lang.) 

89% 

SAMPLE  NOT  ACCESSED  - 42 

A minimum  of  three  calls  were  made  to  a household  before  classifying  it  as  a "no 
answer". 

Refers  to  contacts  made  where  the  telephone  listing  or  respondent  did  not  represent 
a valid  member  of  the  population 

Refers  to  the  number  of  telephone  numbers  in  the  sample  that  could  not  be 
contacted  even  after  five  callbacks 

Refers  to  the  number  of  people  contacted  who  were  qualified  but  who  did  not  want 
to  participate  in  the  survey. 
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COMPLETION  RATE  - STUDENTS  (School  4)  - TELEPHONE  SURVEY 


■ TOTAL  NUMBER  OF  CALLS 

214 

100.00% 

■ Non-Valid/  Non-residential/  household  not  eligible1 

- Fax/Modem/cellular/pager  listings 

- Business  listings 

- Wrong  numbers 

- Duplicate  Numbers 

- Non  Qualified  respondents 

- Other 

8 

3.74% 

■ Numbers  not  in  Service 

6 

2.80% 

SUBTOTAL 

14 

6.54% 

■ NEW  BASE 

200 

93.46% 

■ No  Answer/Busy/Respondent  not  Available2 

- Answering  Machines 

- Voice  Mail 

- Call  Backs 

83 

38.79% 

■ Communication/Language  Barrier 

1 

0.47% 

■ Incomplete  Interviews 

- Respondent  Terminated 

- Quota  Filled  (gender,  geographic  location  etc. 

■ Refusals3 

14 

6.54% 

SUBTOTAL 

98 

45.79% 

NET  COMPLETIONS 

102 

47.66% 

COMPLETION  RATE: 

Completes  / New  Base  - (No  Answer  + Comm.  Lang.) 

87.93% 

A minimum  of  three  calls  were  made  to  a household  before  classifying  it  as  a "no 
answer". 

Refers  to  contacts  made  where  the  telephone  listing  or  respondent  did  not  represent 
a valid  member  of  the  population 

Refers  to  the  number  of  telephone  numbers  in  the  sample  that  could  not  be 
contacted  even  after  five  callbacks 

Refers  to  the  number  of  people  contacted  who  were  qualified  but  who  did  not  want 
to  participate  in  the  survey. 
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COMPLETION  RATE  - TEACHERS  (School  1)  - TELEPHONE  SURVEY 


■ TOTAL  NUMBER  OF  CALLS 

21 

100.00% 

■ Non-Valid/  Non-residential/  household  not  eligible1 

- Fax/Modem/cellular/pager  listings 

- Business  listings 

- Wrong  numbers 

- Duplicate  Numbers 

- Non  Qualified  respondents 

- Other 

1 

4.76% 

■ Numbers  not  in  Service 

1 

4.76% 

SUBTOTAL 

2 

9.52% 

■ NEW  BASE 

19 

90.48% 

■ No  Answer/Busy/Respondent  not  Available2 

- Answering  Machines 

- Voice  Mail 

- Call  Backs 

3 

14.29% 

■ Communication/Language  Barrier 

- 

■ Incomplete  Interviews 

- Respondent  Terminated 

- Quota  Filled  (gender,  geographic  location  etc. 

■ Refusals3 

2 

9.52% 

SUBTOTAL 

5 

23.81% 

NET  COMPLETIONS 

14 

66.67% 

COMPLETION  RATE: 

Completes  / New  Base  - (No  Answer  + Comm.  Lang.) 

87.50% 

A minimum  of  three  calls  were  made  to  a household  before  classifying  it  as  a "no 
answer". 

Refers  to  contacts  made  where  the  telephone  listing  or  respondent  did  not  represent 
a valid  member  of  the  population 

Refers  to  the  number  of  telephone  numbers  in  the  sample  that  could  not  be 
contacted  even  after  five  callbacks 

Refers  to  the  number  of  people  contacted  who  were  qualified  but  who  did  not  want 
to  participate  in  the  survey. 
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COMPLETION  RATE  - TEACHERS  (School  2)  - TELEPHONE  SURVEY 


■ TOTAL  NUMBER  OF  CALLS 

41 

100.00% 

■ Non-Valid/  Non-residential/  household  not  eligible 

- Fax/Modem/cellular/pager  listings 

- Business  listings 

- Wrong  numbers 

- Duplicate  Numbers 

- Non  Qualified  respondents 

- Other 

1 

2.44% 

■ Numbers  not  in  Service 

4 

9.76% 

SUBTOTAL 

5 

12.20% 

■ NEW  BASE 

36 

87.80% 

■ No  Answer/Busy/Respondent  not  Available2 

- Answering  Machines 

- Voice  Mail 

- Call  Backs 

23 

56.10% 

■ Communication/Language  Barrier 

- 

■ Incomplete  Interviews 

- Respondent  Terminated 

- Quota  Filled  (gender,  geographic  location  etc. 

■ Refusals3 

1 

2.44% 

SUBTOTAL 

24 

58.54% 

NET  COMPLETIONS 

12 

29.27% 

COMPLETION  RATE: 

Completes  / New  Base  - (No  Answer  + Comm.  Lang.) 

92.31% 

A minimum  of  three  calls  were  made  to  a household  before  classifying  it  as  a "no 
answer". 

Refers  to  contacts  made  where  the  telephone  listing  or  respondent  did  not  represent 
a valid  member  of  the  population 

Refers  to  the  number  of  telephone  numbers  in  the  sample  that  could  not  be 
contacted  even  after  five  callbacks 

Refers  to  the  number  of  people  contacted  who  were  qualified  but  who  did  not  want 
to  participate  in  the  survey. 
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COMPLETION  RATE  TEACHERS  (School  3)  - TELEPHONE  SURVEY 


■ TOTAL  NUMBER  OF  CALLS 

30 

100.00% 

■ Non-Valid/  Non-residential/  household  not  eligible1 

- Fax/Modem/cellular/pager  listings 

- Business  listings 

- Wrong  numbers 

- Duplicate  Numbers 

- Non  Qualified  respondents 

- Other 

■ Numbers  not  in  Service 

- 

SUBTOTAL 

- 

■ NEW  BASE 

30 

100.00% 

■ No  Answer/Busy/Respondent  not  Available2 

- Answering  Machines 

- Voice  Mail 

- Call  Backs 

4 

13.33% 

■ Communication/Language  Barrier 

- 

■ Incomplete  Interviews 

- Respondent  Terminated 

- Quota  Filled  (gender,  geographic  location  etc. 

■ Refusals3 

1 

3.33% 

SUBTOTAL 

5 

16.67% 

NET  COMPLETIONS 

25 

83.33% 

COMPLETION  RATE: 

Completes  / New  Base  - (No  Answer  + Comm.  Lang.) 

96.15% 

A minimum  of  three  calls  were  made  to  a household  before  classifying  it  as  a "no 
answer". 

Refers  to  contacts  made  where  the  telephone  listing  or  respondent  did  not  represent 
a valid  member  of  the  population 

Refers  to  the  number  of  telephone  numbers  in  the  sample  that  could  not  be 
contacted  even  after  five  callbacks 

Refers  to  the  number  of  people  contacted  who  were  qualified  but  who  did  not  want 
to  participate  in  the  survey. 
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COMPLETION  RATE  - TEACHERS  (School  4)  - TELEPHONE  SURVEY 


■ TOTAL  NUMBER  OF  CALLS 

32 

100.00% 

■ Non-Valid/  Non-residential/  household  not  eligible1 

- Fax/Modem/cellular/pager  listings 

- Business  listings 

- Wrong  numbers 

- Duplicate  Numbers 

- Non  Qualified  respondents 

- Other 

2 

6.25% 

■ Numbers  not  in  Service 

1 

3.13% 

SUBTOTAL 

3 

9.38% 

■ NEW  BASE 

29 

90.63% 

■ No  Answer/Busy/Respondent  not  Available2 

- Answering  Machines 

- Voice  Mail 

- Call  Backs 

5 

15.63% 

■ Communication/Language  Barrier 

■ Incomplete  Interviews 

- Respondent  Terminated 

- Quota  Filled  (gender,  geographic  location  etc. 

■ Refusals3 

1 

3.13% 

SUBTOTAL 

6 

18.75% 

NET  COMPLETIONS 

23 

71.88% 

COMPLETION  RATE: 

Completes  / New  Base  - (No  Answer  + Comm.  Lang.) 

95.83% 

A minimum  of  three  calls  were  made  to  a household  before  classifying  it  as  a "no 
answer". 

Refers  to  contacts  made  where  the  telephone  listing  or  respondent  did  not  represent 
a valid  member  of  the  population 

Refers  to  the  number  of  telephone  numbers  in  the  sample  that  could  not  be 
contacted  even  after  five  callbacks 

Refers  to  the  number  of  people  contacted  who  were  qualified  but  who  did  not  want 
to  participate  in  the  survey. 
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Appendix  B:  Manpower  Requirements  and  Key  Tasks 


Environics  West 


The  following  chart  identifies: 

1.  The  number  of  hours  required  to  enter  the  data  into  computer  files  for  self- 
completed  surveys. 

2.  The  number  of  hours  required  to  conduct  the  telephone  interviews,  including 
supervision  using  a CATI  system  (e.g.,  simultaneous  collection  and  data  entry) 
versus  not  using  CATI  (e.g.,  conducting  the  interviews  on  paper  and  then  inputting 
the  data  into  computer  file  (e.g.,  for  telephone  surveys). 


Hours  Incurred  to  Complete  Data  Entry  of  Self  Completed  Surveys  and 
Completion  of  Interviews  for  the  Telephone  Surveys 


Location 

Self  Completed 

Telephone 

# of  Completes 

Hours 
Incurred 
to  enter 
the  data 

# of  Completes 

Hours 

Incurred 

Interview 

using 

CATI 

Hours 
Incurred  if 
Telephone 
Interviews 
were  not 
conducted  by 

Parents 

Teachers 

Students 

Parents 

Teachers 

Students 

School  1 

18 

20 

66 

7.00 

100 

14 

100 

99.00 

fu&bb 

School  2 

128 

32 

178 

28.50 

100 

12 

120 

126.25 

154.75 

School  3 

62 

27 

161 

12.25 

100 

25 

101 

127.50 

139.75 

School  4 

279 

30 

414 

22.75 

103 

23 

102 

111.75 

134.50 

Note:  There  were  an  additional  15  hours  per  location  for  sample  input,  coding,  editing, 
administration,  and  table  creation.  All  hours  include  supervision. 


In  order  to  understand  better  the  manpower  involved,  it  should  be  noted  that  all  Environics 
staff  members  are  professionally  trained  for  both  telephone  interviewing  and  data  input  of 
self- administered  surveys  into  the  CATI  system.  Therefore,  it  should  be  presumed  that  any 
personnel  (at  the  school  board  level)  would  require  initial  training  on  interviewing 
techniques  and  software  experience  prior  to  commencing  the  field  work  associated  with  such 
a project. 

Most  Environics  input  staff  work  a six  hour  shift  per  day.  Given  this  scenario,  one 
individual  would  be  required  for  between  one  and  five  days  to  enter  all  received  self- 
administered  surveys. 

Professionally  trained  telephone  interviewers  work  approximately  4.5  hours  per  evening  shift 
(5:00  p.m.  to  9:30  p.m.).  In  this  scenario,  approximately  four  to  five  interviewers  would 
be  required  over  a five  day  period.  Also,  supervision  hours  would  be  added,  and  these 
hours  would  require  one  individual  for  the  duration  of  field  telephone  field  activities 
(between  20  and  25  hours  per  location). 
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